In this session, we will learn about variables, groups, stimuli design and counterbalancing, all of them are crucial aspects of experimental design,
There are two types of variables that we need to consider:
Independent Variable (IV): the independent variable affects some behavior or outcome. The independent variable defines groups across which we look for differences, and we can intentionally manipulate these groups to test our hypothesis. The independent variable in this case is the quantifier (condition).
Dependent Variable (DV): Dependent variables are the outcome measures of the experiment – their value depends on the independent variables – and are specific, measurable behaviors (e.g., RTs, acceptability measures)
Within-subject variables are manipulated by testing each participant at each level of the variable, participants see all conditions.
Between-subject variables are manipulated by testing different participants at each level, group 1 sees conditions 1, and group 2 sees condition 2.
You experiment is only as good as your stimuli are; this is why creating stimuli is usually the longest and most arduous part of designing an experiment.
Because we are taking a hypothesis-driven approach, we will design our stimuli based on the experimental manipulation we want to investigate.
Here is a real life example:
A substantial body of research has shown that, in ambiguous relative clauses containing the preposition with, disambiguation is preferred towards the second noun phrase (NP2). In the example below, this thematic preposition creates a strong preference to chunk the adjective and NP2 for disambiguation:
Now, in Spanish, we can introduce a second manipulation towards disambiguation using gender:
In the first sentence, the gender manipulation goes in line with the preference for NP2 attachment, while the second sentence violates it
First, what are the conditions? + Conditions + NP2 Masculine + NP2 Feminine + NP1 Masculine + NP1 Feminine
What confounding factors might we need to take into consideration when creating the stimuli? Confounds occur when the levels of the independent variable vary directly with some other factor that is not of interest to the hypothesis of the study but nevertheless has an impact on the dependent variable.
Answer
Correct, some examples are:
Let’s see another real life example:
As you can see, the nouns (NP1 vs. NP2) in this list have columns with attributes such as the ones we mentioned above. The colors blue and green represent matching because these nouns were matched by list, what does this mean?
Before moving on to other aspects of experimental design it is important to consider the following question: How many sentences per condition should I have?
The general rule of thumb is:
In addition, you want at least 30 participants, unless you’re doing a production study.
Counterbalancing is crucial in repeated measures/within-subjects designs because participants are seeing the same conditions multiple times (although the items are different).
While counterbalancing will not completely eliminate confounds like order effects, it does help distribute them evenly across all experimental conditions such that their influence is balanced and does not confound the main effects due to the independent variables.
Thus, counterbalancing is meant to control:
Counterbalancing can be obtained through different designs, the most common in Linguistics is within-subjects counterbalancing. This can be carried out through a Latin Square or complete counterbalancing:
A Latin square is a type of permutation. Permutation involves deriving all possible combination of a sequence. If we start with the sequence ABCD (x4) this would yield 4 possible permutations (for a total of 16 items, 4x4):
Let’s see a real-life example:
Because we have 4 conditions in the current experiment, this yields a total of 80 stimuli (4x20).
We will create 4 lists of counterbalanced materials. Each subject will be assigned to an alternating list (e.g., subject 1=list 1, subject 2=list 2, and so on).
We have created a table were order and lists are columns.
Then we assign each batch of n=20 stimuli to an order in each list alternating the starting point (see List 1 = ABCD). In doing so, we obtain all possible permutations of items per condition, per list, such that a participant never sees the same item in more than one condition.
NOTE: implementing Latin Squares in experiments with an odd number of conditions is quite challenging.
This is another form or simple permutation, it is easier than Latin Squares and can be easily implemented in experiments with odd number of conditions.
In this case, we have created a table were conditions are columns and lists are rows.
Then we assign each batch of n=20 stimuli to a condition in each list alternating the starting point (see column A). In doing so, we also obtain all possible permutations of items per condition and per list, such that a participant never sees the same item in more than one condition.
We use this method to deal with variation that is not inherent to our sample but, rather, a by-product of our experimental design (mostly). By eliminating this variation we ensure that the results we observe are indeed cause by the effect of our experimental manipulation.
Examples of the kind of variation randomization helps deal with:
There are two main types of randomization:
True randomization: entails obtaining random numbers and assigning random numbers to each subject or conditions.
Pseudo-randomization: the “randomization” has a systematic order provided by the reviewers (e.g., a filler trial will appear every 10 experimental trials).
This corresponds to Andy Fields’ book Chapter 1.6.3
Research methods
Randomization